1. The Field of the Invention
The present invention relates to software development technology. More specifically, the present invention relates to mechanisms for testing an object to identify combinations of input values that expose failures in the object's output.
2. Background and Related Art
Computers have revolutionized the way we work and play. There are an enormous variety of functions and applications that may be implemented by a general purpose computing system in response to the execution of a software application. The utility and functionality of the computing system does, however, rely on the proper coding of the source code that was compiled or interpreted into the binary instructions that are actually executed by the processor. If there is a coding error, this will often result in a deviation from expected functionality.
Extraordinary efforts are currently in place to reduce the number of unexpected performance deviations in many software programs before and after the software application are shipped to market. However, the creativity of software programmers and designers has led to increasingly complex and powerful software applications. As the complexity of the software application increases, so often does the number of lines of source code needed to generate the software application.
One way of drafting source code in a more organized manner, is to use object-oriented programming to cause run-time objects to be generated in memory. These objects have associated methods (which are potentially also objects) that can be invoked to manipulate (e.g., create, destroy, perform operations on) the object associated with the method or even other objects. A method is typically invoked by supplying an appropriate input value to each input field (also viewed as objects) of the method (commonly referred to as “calling” the method). Accordingly, a method can also manipulate the objects that are passed to a method when the method is invoked. For example, to create a “FileStream” object, a File Stream creation method can be called with appropriate input values for a File Mode input field, a File Access input field, and a File Share input field. Depending on the input values supplied to these input fields, a FileStream object with particular characteristics can be created.
Unfortunately, even when a method is properly invoked, the method may encounter a situation that it cannot address. For example, some combinations of supplied input values may cause a method to generate inappropriate output or even cause the method to fail to generate any output at all. Failure in a method's output can result, for example, when a supplied input value is incompatible with an operating system (e.g., supplying a filename including unsupported characters or an invalid path), is of an inappropriate data type (e.g., supplying a floating point number to a Boolean input field), or when a first input value is inconsistent with a second input value. Referring back to the File Stream creation method, it would be inconsistent to supply an input value of “READ” to the File Access input field (indicating read-only access) and an input value of “APPEND” to the File Mode input field (thereby implying the file stream will be created for writing). When a method is supplied inconsistent input values the method will typically fail and return an error message, or “throw an exception.”
Accordingly, methods (as well as other objects) in a software application are often tested to attempt to identify combinations of input values that will cause output failures. One conventional method of testing software is to have a human tester (which may or may not be the programmer) manually analyze a method (or specification for the method) to identify input fields for the method. The tester may also be required to manually generate rules associated with the input fields, such as, for example, a rule restricting combinations of input values that would result in creating (or opening) a File Stream object in append mode with read-only access. Manually analyzing a method (or specification) and manually generating rules is time consuming and can introduce human error into the testing process. For example, if a tester identifies an inappropriate input field or misspells text included in a rule, the validity of test results can be affected. Further, the time consumed to analyze and generate rules for a number of interrelated methods, such as, for example, those of a software module or application, can be quite significant as each method must be individually analyzed and different rules manually generated for each method.
After identifying input fields and generating associated rules, the tester converts the identified input fields and associated rules into an appropriate format (e.g., a text-based format) that is compatible with a test case generator (potentially introducing additional human error into the testing process). The appropriately formatted input fields and associated rules are then used as input to the test case generator. Based on the input fields and associated rules, the test case generator outputs a number of text-based test cases, each text-based test case representing a possible combination of input values that can be supplied to invoke the method.
Unfortunately, even when the number of input fields is relatively small and each input field has a finite number of input values, the number of possible combinations of input values can be relatively large. For example, a method with five input fields, each having five possible input values would have 3,125 (55) different possible combinations of input values. Methods having additional input fields and/or having input fields with more possible input values can have significantly more (potentially millions of) possible combinations of input values. As a result, it may take days or even weeks to execute test cases for every possible combination of input values for a method.
For example, in a testing environment that can execute one test case per second it would take approximately 11 days to execute 1 million test cases. Some testing environments can execute test cases more efficiently (e.g., one test case per millisecond) thereby reducing the overall time needed to test a method. However, when testing a software module or application that includes hundreds or even thousands of methods (as well as potentially thousands of other objects) the time requirements can still be quite large. Thus for efficiency purposes, a tester may limit the number of test cases generated for each method based on generation time and execution time constraints Accordingly, it would not be uncommon for even an experienced tester to miss some of the combinations of input values likely to cause output failure in a method.
After text-based test cases for a method are generated, the tester supplies the text-based test cases as input to a testing framework for the method. For each text-based test case, the testing framework reads in the text-based test case and converts input values from text-based test case into typed input values (of the input field data types in the method being tested). For example, if an input field of the method being tested is to receive an integer value, the test framework converts a text-based representation of an input value for the input field into an integer value. After converting input values, the testing framework executes the text-based test case by invoking the method with the combination of input values read from the text-based test case.
Thus for each method in a software module that is to be tested, the tester (or programmer) must develop a testing framework that can read text-based test cases, convert input values to appropriate input field data types, and properly invoke the method. Since many methods in a software module or application will have different combinations of data types and possible input values, a tester (or programmer) may be required to develop a separate testing framework for each method in a software module or application. Developing separate testing frameworks for each method in a software module or application is time consuming and potentially consumes tester (or programmer) resources that could otherwise be used to correct identified output failures (or develop new software). Accordingly, what would be advantageous are mechanisms for testing software in a more automated manner.